Fix (examples/llm): Fix infinite loop in LLM entrypoint with WikiText2 #1044
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The distribution of the sequence lengths in WikiText2, when fuse_sequences is set to False, is shown in the following picture: |
pablomlago
changed the title
|
Can you update the readme? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix for issue #1036. When fuse_sequences is False, and the seqlen is greater than 842 (maximum sequence length in WikiText2), the condition in line 131 of optimum/amd/brevitas/data_utils.py always evaluates to True, resulting in an infinite loop (see commit d0d06070225804dc6fecde00911ae38b53080123 in optimum-amd).
Consequently, we opted for enabling passing fuse_sequences as an argument and warning the user, when fuse_sequences=False, than an overly large seqlen can result in the infinite loop indicated in issue #1036.